Automated generation of sentence-based descriptors from imaging data

ABSTRACT

A computer-implemented method, a computer system and a non-transitory computer-readable medium for constructing human-readable sentences from imaging data of a subject can include: receiving imaging data including image elements of at least one region of interest of the subject; segmenting the imaging data of the region of interest into a plurality of sub-regions, where each sub-region includes a portion of the image elements; calculating an abnormality factor for each of the sub-regions by quantitatively analyzing segmented image information of the imaging data of the sub-regions using data from a normal database; comparing each abnormality factor to a threshold value; constructing a human-understandable sentence for the subject when a corresponding abnormality factor exceeds the threshold, where each human-understandable sentence references a physical structure threshold associated with the calculation for the region or sub-region; and outputting the human-understandable sentences for the at least one region of the subject.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit to U.S. Provisional Application No.62/246,490, filed on Oct. 26, 2015, the content of which is herebyincorporated by reference in its entirety herein.

GOVERNMENT INTEREST

This invention was made in part with Government Support under NationalInstitutes of Health Grant Nos. EB015909/EB017638/NS084957. TheGovernment has certain rights in the invention.

BACKGROUND

Currently, radiological examinations of brain MR images can be solelybased on subjective judgment utilizing radiologists' knowledge andexperience. The inputs of the process are a set of MR images withdifferent contrasts, such as T1-weighted, T2-weighted,diffusion-weighted, and FLAIR images, and the outputs are free texts.The contents of the texts are typically descriptions of remarkableanatomical features and often, but not always, contain diagnosis basedon such anatomical features. In this process, we can consider the roleof the human is to translate the brain appearance in the MR images toclinically meaningful languages. During this translation, features thatare judged to be within the normal range are filtered out andabnormalities that are visually appreciable and judged to be clinicallyimportant are documented.

The fact that the MR image reading is based on subjective judgment andthe outputs are nonstructurized free texts is often a subject ofcriticisms. The process to convert the anatomical features in the imagesinto the language is not documented and the criteria are vague. The textoutputs are not structured, difficult to search, and hinderpopulation-based analyses. The inter- and intra-rater reliability isalso in question.

In the last two decades, our technologies for quantitative imageanalyses developed significantly, supporting a huge amount of MR-basedbrain research. However, these technologies have been rarely adopted tosupport clinical practice. For example, one of the most commonly usedquantitative analyses is voxel-based analysis, which identifiespotentially abnormal voxels in a fully automated manner. Thesevoxel-by-voxel results are based on standardized anatomical coordinatesand do not carry anatomical meaning or semantic labels. Forinterpretation, human still needs to be involved and, in addition, humanrarely evaluate anatomy in a voxel-by-voxel manner. Thus, the finalinterpretation of the results still relies on examination by human,while human rarely evaluate anatomy in a voxel-by-voxel manner. Thereremain conceptual gaps between the computer-generated results and theway human understand and communicate the anatomical observations.

An alternative approach is to segment MR images into structural unitsand evaluate the properties of the units, such as the volumes. Thisapproach converts the 1-million voxel coordinate information to thevolumes of several hundred structures, which represent anatomicalrepresentations much closer to human's evaluations. However, thesequence of hundreds of numbers is still non-interpretable for thehuman.

Currently, radiological examinations of brain MR images are solely basedon subjective judgment utilizing radiologists' knowledge and experience.The inputs of the process are a set of MR images with differentcontrasts, such as T1-weighted, T2-weighted, diffusion-weighted, andFLAIR images, and the outputs are free texts. The contents of the textsare typically descriptions of remarkable anatomical finding (called“radiological report” hereafter) and often, but not always, containdiagnosis based on the observed anatomical features. In this process, wecan consider the role of the radiologists is to translate the anatomicalfeatures captured in the MR images to clinically meaningful languages(semantic labels). During this translation, features that are judged tobe within the normal range are filtered out and abnormalities that arevisually appreciable and judged to be clinically important aredocumented. One of the most important aspects of this process is that itreduces the voxel-based data in an order of 10 MB into less than 1 KB ofclinically meaningful and human-understandable information. Thisconversion of the high-dimensional imagery to a semantic label is theholy grail of image analysis, where the ability of the human oftenremains unmatched by that of computer algorithms. The conversion byhuman, however, is often criticized in terms of its accuracy andprecision (reproducibility). In the era of modern medical informaticsand Big Data analysis, however, what is probably more problematic is thefact that the thought process of this huge data contraction (in theorder of 104) is not documented and, thus, not available in a readilyusable format. Further, the text outputs are not structured anddifficult to search and analyze. These characteristics of the currentradiological reading hinder large-scale evidence-based analyses. Fromeducational point of views, this also means that the expertise needs tobe taught only through mentoring.

SUMMARY

A computer-implemented method of constructing human-readable sentencesfrom imaging data of a subject can include: receiving imaging datacomprising a plurality of image elements of at least one region ofinterest of the subject; segmenting, using at least one data processor,the imaging data of said region of interest into a plurality ofsub-regions, each sub-region comprising a portion of said plurality ofimage elements; calculating an abnormality factor for each of thesub-regions by quantitatively analyzing segmented image information ofsaid imaging data of said sub-regions using data from a normal database;comparing each abnormality factor to a threshold value; constructing ahuman-understandable sentence for the subject when a correspondingab-normality factor exceeds the threshold, each human-understandablesentence referencing a physical structure threshold associated with thecalculation for the region or sub-region; and outputting thehuman-understandable sentences for the at least one region of thesubject.

A computer system for constructing human-readable sentences from imagingdata of a subject can include: a memory comprising computer-executableinstructions; and a data processor that is coupled to the memory. Thedata processor can be configured to execute the computer-executableinstructions to: receive imaging data comprising a plurality of imageelements of at least one region of interest of the subject; segment,using at least one data processor, the imaging data of said region ofinterest into a plurality of sub-regions, each sub-region comprising aportion of said plurality of image elements; calculate an abnormalityfactor for each of the sub-regions by quantitatively analyzing segmentedimage information of said imaging data of said sub-regions using datafrom a normal database; compare each abnormality factor to a thresholdvalue; construct a human-understandable sentence for the subject when acorresponding abnormality factor exceeds the threshold, eachhuman-understandable sentence referencing a physical structureassociated with the calculation for the region or sub-region; and outputthe human-understandable sentences for the at least one region of thesubject.

A non-transitory computer-readable medium for constructinghuman-readable sentences from imaging data of a subject can includehaving instructions that, when executed by at least one data processor,cause a computing system to: receive imaging data comprising a pluralityof image elements of at least one region of interest of the subject;segment, using at least one data processor, the imaging data of saidregion of interest into a plurality of sub-regions, each sub-regioncomprising a portion of said plurality of image elements; calculate anabnormality factor for each of the sub-regions by quantitativelyanalyzing segmented image information of said imaging data of saidsub-regions using data from a normal database; compare each abnormalityfactor to a threshold value; construct a human-understandable sentencefor the subject when a corresponding abnormality factor exceeds thethreshold, each human-understandable sentence referencing a physicalstructure associated with the calculation for the region or sub-region;and output the human-understandable sentences for the at least oneregion of the subject.

These and other features and advantages will be apparent from a readingof the following detailed description and a review of the associateddrawings. It is to be understood that both the foregoing generaldescription and the following detailed description are explanatory onlyand are not restrictive of aspects as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts a template atlas that defines 200 structures.

FIG. 2 depicts automated segmentation of the brain into 200 structuresby applying the template atlas information.

FIG. 3 depicts a hierarchical definition of the brain structures withdifferent levels of granularity.

FIG. 4 depicts comparison of granularity reduction by isotropicresolution reduction (upper row) and ontology-based structural reduction(bottom row).

FIG. 5A depicts a screenshot of the RoiEditor interface that allowsautomated visualization and quantification of the ontology-basedmulti-granularity image analysis.

FIG. 5B depicts a test-retest analysis of healthy subjects scannedtwice, according to one embodiment.

FIG. 6 shows results of principal component analysis (PCA) using the 254ROIs in the highest granularity level.

FIGS. 7A and 7B show the anatomical variability at two differentgranularity levels (level 1 and 4) of segmentation for the young normaladults.

FIG. 8 shows “classical” view of anatomical abnormalities of the ADpopulation.

FIG. 9 shows an alternative view of the same data at level 5, in whichthe within-group data reduction was not performed and the anatomicalphenotype of each individual is delineated using z-scores.

FIG. 10 shows image-based representation of the multi-granularityanalysis of one PPA patients.

FIG. 11 shows image-based presentation of three representative PPA casesusing T1-superimposition views at level 5, in which the color-codedz-score information is superimposed on their T1 weighted images.

FIG. 12 shows a flowchart of image analysis sentence-based generation,according to an embodiment of the invention.

Additional features, advantages, and embodiments of the invention areset forth or apparent from consideration of the following detaileddescription, drawings and claims. Moreover, it is to be understood thatboth the foregoing summary of the invention and the following detaileddescription are examples and intended to provide further explanationwithout limiting the scope of the invention as claimed.

DETAILED DESCRIPTION

Some embodiments of the current invention are discussed in detail below.In describing embodiments, specific terminology is employed for the sakeof clarity. However, the invention is not intended to be limited to thespecific terminology so selected. A person skilled in the relevant artwill recognize that other equivalent components can be employed andother methods developed without departing from the broad concepts of thecurrent invention. All references cited anywhere in this specification,including the Background and Detailed Description sections, areincorporated by reference as if each had been individually incorporated.

This application discloses methods of generating sentences fromquantitative image analysis results. There are several components.First, clinically important anatomical features can be identified in thequantitative analysis results. Second, a dictionary can be used toconvert the anatomical features captured by the quantitative analysisinto a human language.

Definitions

Abnormality Factor—In some embodiments, the term “abnormality factor”can mean, for example, a physiological or anatomical risk or abnormalityin a subject that can be detected based on size and intensity of imagingdata of the subject. For example, imaging data may indicatethinner/thicker than average anatomical features, smaller or largeranatomical features, and/or brighter/darker images of the anatomicalfeatures than is normal. These abnormalities can allow for a clinicaldetermination of a risk or abnormality in the subject.

Human-Understandable Sentence—A human-understandable sentence can mean,for example, words ordered into phrases, sentences, etc. according torules such as grammatical rules. They can be output and/or displayed inwritten form, sign language and/or verbal sounds, for example, that arecombined in an ordered manner to convey meaning to a human such asnatural language.

The term “computer” or “computer system” is intended to have a broadmeaning that can include computing devices such as, e.g., but notlimited to, standalone or client or server devices. The computer may be,e.g., (but not limited to) a personal computer (PC) system running anoperating system such as, e.g., (but not limited to) MICROSOFT® WINDOWS®NT/98/2000/XP/Vista/Windows 7/8/etc. available from MICROSOFT®Corporation of Redmond, Wash., U.S.A. or an Apple computer executingMAC® OS from Apple® of Cupertino, Calif., U.S.A. However, the inventionis not limited to these platforms. Instead, the invention may beimplemented on any appropriate computer system running any appropriateoperating system. In one illustrative embodiment, the present inventionmay be implemented on a computer system operating as discussed herein.The computer system may include, e.g., but is not limited to, a mainmemory, random access memory (RAM), and a secondary memory, etc. Mainmemory, random access memory (RAM), and a secondary memory, etc., may bea computer-readable medium that may be configured to store instructionsconfigured to implement one or more embodiments and may comprise arandom-access memory (RAM) that may include RAM devices, such as DynamicRAM (DRAM) devices, flash memory devices, Static RAM (SRAM) devices,etc.

The computer may also include an input device that may include anymechanism or combination of mechanisms that may permit information to beinput into the computer system from, e.g., a user. The input device mayinclude logic configured to receive information for the computer systemfrom, e.g. a user. Examples of the input device may include, e.g., butare not limited to, a mouse, pen-based pointing device, or otherpointing device such as a digitizer, a touch sensitive display device,and/or a keyboard or other data entry device (none of which arelabeled). Other input devices may include, e.g., but are not limited to,a biometric input device, a video source, an audio source, a microphone,a web cam, a video camera, and/or other camera. The input device maycommunicate with a processor either wired or wirelessly.

The computer may also include output devices which may include anymechanism or combination of mechanisms that may output information froma computer system. An output device may include logic configured tooutput information from the computer system. Embodiments of outputdevice may include, e.g., but not limited to, display, and displayinterface, including displays, printers, speakers, cathode ray tubes(CRTs), plasma displays, light-emitting diode (LED) displays, liquidcrystal displays (LCDs), printers, vacuum florescent displays (VFDs),surface-conduction electron-emitter displays (SEDs), field emissiondisplays (FEDs), etc. The computer may include input/output (I/O)devices such as, e.g., (but not limited to) communications interface,cable and communications path, etc. These devices may include, e.g., butare not limited to, a network interface card, and/or modems. The outputdevice may communicate with a processor either wired or wirelessly. Acommunications interface may allow software and data to be transferredbetween the computer system and external devices.

The term “processor” or “data processor” is intended to have a broadmeaning that includes, but is not limited to, one or more processors,such as that are connected to a communication infrastructure (e.g., butnot limited to, a communications bus, cross-over bar, interconnect, ornetwork, etc.). The term data processor may include any type ofprocessor, microprocessor and/or processing logic that may interpret andexecute instructions (for example, a field programmable gate array(FPGA)). The data processor may comprise a single device (for example, asingle core) and/or a group of devices (e.g., multi-core). The dataprocessor may include logic configured to execute computer-executableinstructions configured to implement one or more embodiments. Theinstructions may reside in main memory or secondary memory. The dataprocessor may also include multiple independent cores, such as adual-core processor or a multi-core processor. The data processors mayalso include one or more graphics processing units (GPU) which may be inthe form of a dedicated graphics card, an integrated graphics solution,and/or a hybrid graphics solution. Various illustrative softwareembodiments may be described in terms of this illustrative computersystem. After reading this description, it will become apparent to aperson skilled in the relevant art(s) how to implement the inventionusing other computer systems and/or architectures.

The term “data storage device” is intended to have a broad meaning thatincludes a removable storage drive, a hard disk installed in hard diskdrive, flash memories, removable discs, non-removable discs, etc. Inaddition, it should be noted that various electromagnetic radiation,such as wireless communication, electrical communication carried over anelectrically conductive wire (e.g., but not limited to twisted pair,CATS, etc.) or an optical medium (e.g., but not limited to, opticalfiber) and the like, may be encoded to carry computer-executableinstructions and/or computer data that embodiments of the invention one.g., a communication network. These computer program products mayprovide software to the computer system. It should be noted that acomputer-readable medium that comprises computer-executable instructionsfor execution in a processor may be configured to store variousembodiments of the present invention.

Embodiments of the invention can include several components including amulti-atlas image segmentation tool, a brain atlas library, a databaseof the normal brains, and a dictionary that translates the quantitativeanalysis results to human-readable sentences.

An aim of embodiments of the invention is to develop a tool thatautomatically performs the translation of MR images into radiologicalreports. We used a state-of-the-art multi-atlas brain segmentation tooland an atlas library with more than 80 fully segmented atlases forhighly robust and accurate segmentation of T1-weighted images. Weemployed a unique scheme to generate anatomical labels, which simulatesthe way radiologists evaluate the brain anatomy. (Djamanakova et al.,2014; Wu et al., 2015) This procedure generates the volumes of 498anatomical labels, which was first applied to data from 238 normalsubjects and the age-dependent normal value of each label was defined.Then the same procedure was applied to the images of 100 dementiapatients acquired in Johns Hopkins University. Based on the normaldatabase, the age-corrected z-score was calculated for each anatomicallabel. We then developed a dictionary to convert the 498 z-scores tohuman-interpretable sentences. For accuracy testing, three radiologistsindependently evaluated the same 100 images and the agreement wasobserved. Although this approach can be potentially applied to bothmorphological (volume) and signal-intensity abnormalities, theevaluation in this paper was limited to the morphological abnormalitiesbased on T1-weighted images. Our primary goal was to measure theaccuracy of the generated sentences. However, during the course of theresearch, we encountered unexpected, yet interesting difficulties inperforming such translation and evaluation. It is our goal not only toreport the performance of our tool, but also to share our experiencesabout the potential problems and difficulties encountered, as well asdiscussions about the possible future directions.

Methods:

Quantitative analysis results: To simplify the method description, brainMRI data can be analyzed, such as T1-weighted images, but the sameapproach can be applied to other organs and other imaging modalities.With 1 mm resolution, a typical brain with more than 1 liter of volumecontains more than 1 million voxels. Brain regional size information,such as atrophy, can be quantitatively mapped to each voxel. Forexample, a brain can be warped to a reference brain and such warping mayexert shrinkage or enlargement of a voxel of interest, givingquantitative information about the voxel-based atrophy or hypertrophystates. By grouping the voxels into structural representation, the1-million localization information could be reduced to, say, 100representative brain structures. Then the anatomical states of the braincan be represented by 100 numbers. In a previous applicationPCT/US14/69161, the content of which is hereby incorporated herein inits entirety, methods are disclosed to evaluate the brain anatomy frommultiple granularity levels by formalizing this type of multiplehierarchical relationships. Namely, the minimum unit of imaging is onevoxel. By grouping multiple voxels, we can generate basic structuralunits. Then, by combining the basic structural units, superstructurescan be generated. Based on pre-determined hierarchical relationships,the anatomy of the brain can be quantitatively analyzed at multiplegranularity levels.

Human perception: When humans evaluate the brain anatomy, they usuallydo not evaluate at each voxel level. As a matter of fact, theydynamically shift the granularity levels between a very macroscopic viewsuch as the features of the entire brain and a microscopic view based ona structural unit that can be identified by the given image contrasts.Segmenting the same brain at several different granularity levels isthus important to mimic the way humans see the anatomy. The generationof a dictionary is much more feasible if the anatomical information isfurther reduced to a smaller representation such as a structure-basedrepresentation. The multi-granularity data can further enable moresophisticated dictionary creation.

Conversion of the quantitative numbers to human-readable sentences:Knowledge-driven approach: In this approach, we will rely on experiencesand knowledge of experienced radiologists to systematically convert thequantification results to sentences. Namely, we first define importantanatomical features experienced radiologists would evaluate. For examplea global brain tissue atrophy is important information they can firstcheck.

Then, the dictionary contains a sentence such as “The brain has (severe,apparent, mild) global atrophy” and the corresponding quantitativeresults would be “the volume of the two hemisphere” or “the volume ofthe two hemisphere/(the volume of the two hemisphere+the volume of theCSF space)”, while the latter contains a normalization factor.Similarly, a sentence, “the patient has left dominant atrophy”, isgenerated when a ratio of “the volume of the left hemisphere/the volumeof the right hemisphere” is smaller than a certain threshold.

Conversion of the quantitative numbers to human-readable sentences:Data-driven approach: A dictionary can also be created based on adata-driven approach. In this case, we can perform a certain type ofcorrelational analysis between the anatomical features captured by aquantitative analysis and certain labeling of the patients such asdiagnosis. If this analysis identifies a correlation between specificanatomical patterns and diseases, a dictionary can be generated. Forexample, relatively smaller sizes of the frontal and temporal lobesimultaneously may trigger a sentence such as “this patient possessesanatomical features typically found in the frontotemporal dementia.”This relationship can be generated not only from the anatomical featuresbut also from non-image clinical information such as neuropsychologicaltesting. A combination of the imaging and non-imaging data can enhancethe accuracy of the dictionary.

Examples of the dictionary:

For example, below an example of multi-granularity report of structuralvolumes is shown;

Granularity Level-1:

Hemisphere_L 532540 mm³ Hemisphere_R 538376 mm³ Brainstem  29551 mm³Cerebellum 122825 mm³ CSF  74772 mm³

From this report, we can test and generate the following sentences basedon the quantitative analysis:

Sentence: “There is (severe, apparent, moderate, mild) globalhemispheric atrophy”Quantitative observation: (Hemisphere L+Hemisphere R)/(the sum of theall structures)

Judgment Criteria:

Z-score of the quantitative observation <5.0: Severe

Z-score of the quantitative observation <4.0: Apparent

Z-score of the quantitative observation <3.0: Moderate

Z-score of the quantitative observation <2.5: Mild

Similarly, ratios of specific structural volumes can also be used as aquantitative measurement and the judgment criteria:

Sentence: “There is (severe, apparent, moderate, mild) asymmetry of thebrain”Quantitative observation: (Hemisphere_U/Hemisphere_R)

Judgment Criteria:

Z-score of the quantitative observation <5.0: Severe

Z-score of the quantitative observation <4.0: Apparent

Z-score of the quantitative observation <3.0: Moderate

Z-score of the quantitative observation <2.5: Mild

In these examples, the judgment criteria were based on z-score,calculated from averages and standard deviations of age-matched controlsubjects. The raw volumes and ratios may also be used.

In the finer granularity levels, more location-specific descriptions canbe generated. For example, Granularity level-3 (numbers are all in mm3):

Frontal_L 137029 Frontal_R 135285 Parietal L 87795 Parietal R 85885Temporal_L 93659 Temporal_R 101828 Limbic L 35872 Limbic R 38737Occipital_L 68287 Occipital_R 63856 Insula L 6604 Insula_R 7598BasalGang_L 11083 BasalGang_R 11263 Thalamus_L 5231 Thalamus R 5692BasalForebrain L 3064 BasalForebrain R 2666 midbrain L 4723 midbrain R4671 Cerebellum_R 61869 Cerebellum_L 60956 Pons_L 6773 Pons_R 7774Medulla_L 2609 Medulla_R 3001 AnteriorWM L 31975 AnteriorWM R 33122PosteriorWM L 6815 PosteriorWM R 6919 CorpusCallosum_L 11616CorpusCallosum_R 12809 InferiorWM L 27129 InferiorWM R 25195 LimbicWM L6381 LimbicWM R 7521 LateralVentricle L 6834 LateralVentricle R 5666 IIIventricle 1718 FrontSul_L 14486 FrontSul_R 9811 CentralSul_L 2291CentralSul_R 1820 TempSul_L 2191 TempSul_R 1963 ParietSul L 7877Pariet5ul R 8289 CinguSul_L 3490 CinguSul_R 2775 OcciptSul_L 1638OcciptSul_R 1962 IV ventricle 1961Sentence: “There is (severe, apparent, moderate, mild) atrophy of theleft frontal lobe”Quantitative observation: (Frontal L)/(the sum of the all structures)

Judgment Criteria:

Z-score of the quantitative observation <5.0: Severe

Z-score of the quantitative observation <4.0: Apparent

Z-score of the quantitative observation <3.0: Moderate

Z-score of the quantitative observation <2.5: Mild

While these dictionaries are used for anatomical descriptions, moreadvanced clinical interpretations such as potential link to specificdiagnosis may also be possible. For example,

Sentence: “The patient's anatomy contains anatomical features (strongly,moderately, mildly) related to the frontotemporal dementia”Quantitative observation: (Frontal L+Frontal R+Temporal L+TemporalR)/(the sum of the all structures)

Judgment Criteria:

Z-score of the quantitative observation <5.0: Strongly

Z-score of the quantitative observation <4.0: Moderately

Z-score of the quantitative observation <3.0: Mildly

Methods and Materials

The overall theory for image-sentence conversion:

In the field of computer vision, brute-force methods are a potentialsolution often resorted to, in which every single voxel of images isexamined in a training set and the knowledge is applied to detect anobject of interest in test images. For brain MRI, voxel-based analysisbelongs to this category. This is a data-driven approach, which coulddiscover new finding that could not be perceived by human eyes. However,this approach, in general, suffers from two types of difficulty. First,many types of modern imaging often have too many voxels to examine,while the majority of the voxels may not be related to importantobservations. This could statistically overwhelm the efficacy of ourjudgment, which is widely recognized as “curse of dimensionality.” Datareduction is, thus, essential. The second problem is that the finalresults, which are often scattered clusters of voxels with statisticalsignificance, may not be anatomically interpretable and difficult torelate them to biological meaning or subsequent judgments in our reallife.

The alternative approach, which was employed in this study, isknowledge-driven. We learned that several levels of knowledge are neededfor the translation. In the first level, we need to decide how multiplevoxels are aggregated to create anatomically meaningful labels. Based onT1-weighted brain MRI, it is possible to discern approximately 300anatomical structures that have been anatomically recognized in thepast. While some of them, such as the hippocampus, are of greatimportance of radiologists, they rarely examine all 300 structures asindependent entities. Rather, if we go over radiological reports,structural definitions such as “tissue”, “parenchyma”, “hemisphere”,“lobes”, and “ventricles”, are far more commonly used. We recognized theinformation reduction of voxel-based data (more than 7 million voxels inthe standard MNI coordinates) to clinically meaningful structuralrepresentation is the first crucial step to understand how humanperceive the anatomy, which we call “Anatomical Knowledge Filter (AKF)”,hereafter. Technically, this can be achieved by automated whole-brainsegmentation and the AKF must be explicitly written as a format ofpre-segmented brain atlases.

The second filter is “Judgment Filter (JF)”, which decides abnormalregions. This requires knowledge about the range of normal.Statistically, this can be simple t-tests or z-scores if data aboutage-matched controls are available. While the AKF achieves informationreduction in the order of 105 (from voxels to structures), radiologistscertainly do not evaluate the 300 structures independently. Somestructures are more clinically important than others and some structuralunits may never been reported as a target of independent evaluation. Wetherefore can apply “Clinical Knowledge Filter (CKF)” to further reducethe information size. The CKF is not simply to create a short list ofimportant structures because it also contains relational filters such as“left-dominant” or “frontal and temporal lobe” atrophy, which requirescomparison with anatomical counterparts.

By moving one step further, the patterns (the combination of multiplestructures) of the abnormality could be related to certain diseases orfunctional outcomes if such databases are available. The fourth filteris the dictionary, which is related to the CKF and triggered based onthe results of the JF to generate human-understandable sentences.

Anatomical Knowledge Filter by the multi-atlas segmentation: The stepsto generate sentences are shown in FIG. 12. The anatomical knowledgeabout the locations and features of the structures were represented bypre-segmented brain atlases. By warping the atlases to individualpatient images, consistent anatomical criteria can be applied to alldata. In this study, we used a modern multi-atlas approach for thesegmentation, in which more than 80 atlases were used. These atlaseswere warped to individual patient images, followed by an arbitrationprocess to reach the final segmentation results. For the multi-atlassegmentation, we used the MriCloud (www.mricloud.org) pipeline, which isbased on a method described in our previous publication (Tang et al.,2015). Briefly, the image warping was performed using non-lineartransformation based on Large Deformation Diffeomorphic Metric Mapping(LDDMM)(Joshi and Miller, 2000), and the following arbitration used theatlas-fusion algorithm (Tang et al., 2015).

While the disclosure refers to embodiments of the invention as being acomputer-implemented method, it is to be understood that embodiments ofthe invention also interchangeably relate to computer systems andnon-transitory computer readable media.

Thus, in some embodiments, a computer-implemented method of constructinghuman-readable sentences from imaging data of a subject can include:receiving imaging data comprising a plurality of image elements of atleast one region of interest of the subject; and segmenting, using atleast one data processor, the imaging data of the region of interestinto a plurality of sub-regions, each sub-region comprising a portion ofthe plurality of image elements. The segmenting can include segmentingthe imaging data of the region of interest into a plurality ofsub-regions at a plurality of levels of granularity, the plurality oflevels of granularity having a relationship such that a level ofgranularity has fewer structures at a lower level of granularity, andwherein the calculating includes calculating an abnormality factor ateach of the plurality of levels of granularity.

The calculating the abnormality factor can include calculating anabnormality factor for each sub-region that exceeds the threshold.

The calculated abnormality factor can be based on calculatingstatistical significance from averages and standard deviations ofage-matched control subject data.

The image information can be at least one of size or intensity of theimaging data. The quantitatively analyzing the image information caninclude measuring differences between the size or intensity of theimaging data of the at least one region with reference imaging data.

The comparing can take place based on a predetermined relationshipbetween a size or intensity of the imaging data and a clinical diagnosisfalling within a statistically significant range. The comparing can takeplace based on non-image clinical information.

The method can further include for the outputted sentences that have aclinically meaningful significance, reconstructing a relationshipbetween the clinically meaningful sentences and the global and segmentedimage information. The relationship can be based on the sizes and/orintensities of a single structure or combinations of multiple segmentedstructures. The sizes and/or intensities of multiple structures can becombined by Boolean and/or arithmetic operators to construct anelaborated relationship between the outputted sentences and anatomicalfeatures. The relationship between the anatomical features and theoutputted sentences can be further elaborated by segmenting the imagingdata at a plurality of levels of granularity, the plurality of levels ofgranularity having a relationship such that a level of granularity hasfewer structures at a lower level of granularity, and wherein thecalculating includes calculating an abnormality factor at each of theplurality of levels of granularity.

The method can further include: mapping a plurality of abnormalityfactors to a plurality of predetermined clinical diagnoses in a databaseon a data storage device; and providing a clinical diagnosis of thesubject based on a correlation between the stored clinical diagnoses andthe outputted sentences of the subject.

The method can further include calculating a global abnormality factorfor the imaging data of the at least one region of interest byquantitatively analyzing global image information of the imaging data ofthe at least one region of interest. The method can include cataloguingthe compared abnormality factor and the global abnormality factor of thesubject based on the calculating steps according to one of a pluralityof predefined severity thresholds. The global abnormality factor can bethe size of an entire organ, such as a brain, and comparing it to apredetermined threshold using, for example, a normal database. Thecalculating the global abnormality factor can include warping theimaging data to reference imaging data and calculating a difference.

The atlases were from the JHU multi-atlas library that identifies 286anatomical structures. Although the conversion from more than 7 millionvoxels to the 286 structures is a huge amount of data reduction,comparing with the structural units often used in radiological reports,the granularity of these definitions are still too fine for most cases.For example, radiologists may describe, “volume loss of the lefttemporal lobe”, but among the 286 structural definitions, there is noentity that corresponds to the name “temporal lobe.” To generate anentity that corresponds to the “temporal lobe”, we need to add sixcortical areas and six peripheral white matter regions that belong tothe temporal lobe. In this manner, for each anatomical name that refersa specific brain location, we need to define the correspondinganatomical definitions by combining the 286 structural elements. Forthis end, we adopted a flexible granularity control tool described inour previous publication (ref). Briefly, multiple levels ofsuperstructures were created based on ontology-based hierarchicalrelationships and applied to the 286 structures. For example, in one ofthe relationships, Level 1 defines only the right and left hemispheres,the brainstem, the cerebellum, and the CSF space. At Level 3, thehemispheres were divided into the frontal, parietal, occipital,temporal, and limbic areas, allowing finer anatomical evaluations. Withall five levels combined, 498 structures were defined. In the aboveexample, the multi-atlas segmentation was used but the segmentationcould be achieved by other techniques such as those using a single atlasor population-based atlases.

Judgment of abnormality: As shown in FIG. 12, the second step is todefine normal ranges for all 498 structures defined in the previousstep. For all defined structures, the normal values were defined basedon the 238 normative data described above and the age-corrected averageand standard deviations were calculated, from which z-scores (=(measuredvolume−age-matched average volume)/standard deviations) were calculatedfor each defined structure. Thus, the method can include calculating anabnormality factor for each of the sub-regions by quantitativelyanalyzing segmented image information of the imaging data of thesub-regions using data from a normal database and comparing eachabnormality factor to a threshold value. The threshold value can be asize, a volume, a distance, or other type of measurement. In the aboveexample, z-scores were used for the judgment of the abnormality, butother statistical methods such as 95% reliability range, t-test, ANOVA,can be used.

Clinical knowledge filter, dictionary, and triggering: Once the 498structures were defined, it was possible to report the volumes and zscores to radiologists, but it would certainly not useful for routineclinical support. From clinical points of views, not all structural areequal. The first role of the clinical knowledge filter is to select thestructures from the 498 defined labels which are believed to beclinically important. In Appendix I, the 38 selected structures in thisstudy are highlighted. The second role is to define the relationships ofthese selected structures which are believed to be important. Theseclinically important relationships can be explicitly defined as a table,which is shown in Appendix 2.

In the final step of the study, the each relationship defined in theclinical knowledge filter was related to specific a specific sentencewith a triggering criteria. For example, a sentence is generated basedon the z-score of the hippocampus with a rule;

-   -   If z-score (hippocampus_L)<−2.0, then trigger, “Volume loss is        observed in the left hippocampus”

The method can thus include constructing a human-understandable sentencefor each of the catalogued abnormality factors, eachhuman-understandable sentence referencing a physical structure and/orthe severity threshold associated with the calculation for the region orsub-region; and outputting the human-understandable sentences for the atleast one region of the subject.

The method can further include analyzing the compared abnormality factorof the subject according to one of a plurality of predefined severitythresholds. Each of the human-understandable sentence can reference aphysical structure and/or a severity threshold associated with thecalculation for the region or sub-region.

In this study for proof-of-concept, we focus on several pre-selectedanatomical features that are frequently evaluated for dementiapopulations, which are explicitly defined in Appendix II. As describedin the previous section, the role of the clinical knowledge filter isnot only to select a small number of important structures, but also toevaluate their relationships. For example, at ontology Level 1definition, there are two anatomical labels; “hemisphere_L” and“hemisphere_R”. Using these labels, the following sentences aretriggered based on their z-scores;

-   -   If “hemisphere_L”<−2.0 AND “hemisphere_R”<−2.0, then “There is        global hemispheric atrophy”    -   If “hemisphere_L”<−2.0 AND “hemisphere_R”>−2.0, then “There is        left-dominant hemispheric atrophy”

Similarly, lobe-specific atrophy could be an important clue for specifictypes of dementia; the frontotemporal dementia frequently accompaniesatrophy in the frontal and temporal lobes. This would require testing ofrelationship among frontal, parietal, occipital, temporal, and limbiclobes such as;

-   -   If “frontal_L”<−2.0 AND “temporal_L”<−2.0 AND “parietal_L”>−0.2        AND “occipital_L”>−2.0 AND “limbic L”>−2.0 AND “frontal R”>−2.0        AND “temporal R”22 −2.0 AND “parietal R”>−0.2 AND “occipital        R”>−2.0 AND “limbic R”>−2.0, then “There is left-dominant        fronto-temporal specific atrophy”

The human-understandable sentence can be thus constructed using a set ofpredetermined rules based on a relationship between a size of astructure and a size of a corresponding at least one sub-region havingan abnormality factor.

The method can further include generating additional structures byanalyzing multiple levels of granularity for the segmented structures.The calculating the abnormality factors can include calculating anabnormality factor for each of the additional structures.

The method can further include determining clinically relevantstructures using a clinical knowledge database from the segmented andgenerated structures. The constructing the human-understandable sentencecan include incorporating the clinically relevant structures.

The constructing the human-understandable sentence can take into accountrelationships among clinically relevant structures.

In addition, there are nested relationships. For example, if theleft-dominant hemispheric atrophy is found, it can be tested if that isdue to atrophy in specific lobes using the following tests;

-   -   If “Frontal L”<−2.0 AND “temporal_L”>−2.0 AND “parietal_L”>−0.2        AND “occipital_L”>−2.0 AND “limbic L”>−2.0, then “The left        hemispheric atrophy is prominent in the frontal lobe.”

As another example of a nested relationship, if it is determined thatthere is bi-hemispheric atrophy, (“T2_L1_Hemisphere_L” less than −3 AND“T2_L1_Hemisphere_R” less than −3), it can then be determined that theatrophy accompanies sulcus expansion (e.g., if “T1_L2_Sulcus_L” morethan 3 AND “T1_L2_Sulcus R” more than 3).

Evaluation by subjective assessment: The 93 images from the MemoryClinic were read by three neuroradiologists with more than 15 years ofexperience. The images were first read by the radiologists without priorknowledge about the automated analyses. The outcomes were free-textradiological reports. Second, they were presented by the sentencesautomatically generated after the segmentation. For each sentence, ifthey found their reports and automated sentences agree, the case wascounted as “agreed.”

Previous application PCT/US14/69161 discloses Medical imaging such asMRI and CT is playing a crucial role for daily image-based diagnosis inRadiology. The images are currently visually evaluated by trainedphysicians and medical decisions are being made by subjective judgments.Currently computational supports for image reading are used only forlimited tissue areas and a vast majority of the images are evaluatedwithout computational supports.

When physicians evaluate anatomy, they have an ability to dynamicallycontrol the level of anatomical granularity they are inspecting. Thisdisclosure is based on our discovery that this dynamic granularitycontrol is the reason why past computational support could neverapproach human's ability to comprehend anatomy and accurately detectabnormalities in patients.

For example, when a radiologist is reading an MR image of a dementiapatient, the doctor first can evaluate the overall brain atrophy. Inthis case, the size of the entire hemisphere and the ventricles areevaluated. The brainstem and the cerebellum size could also be evaluatedas a clue for hemispheric-specific atrophy or to rule out theinvolvement of the cerebellum. Then the overall status of the cortex,the white matter, and the deep gray matter structures are evaluated. Theinspection continues to smaller granularity levels, in which atrophy ofeach lobes and specific gray matter nuclei are evaluated. For example,the involvement of only the temporal lobe could indicate a specificdisease class. Intensity abnormalities in the white matter could alsoindicate diffuse axonal injuries. The granularity level of the visualinspection could also increase substantially when the doctor is seekingfor a certain type of small anatomical signatures; such as the volumeloss of the caudate in the Huntington's disease or intensity abnormalityin the pons for a certain type of ataxia.

This type of dynamic granularity control has never been implemented anddeployed in the computational diagnosis supports in the past. Forquantitative image analysis, the highest granularity level, which is onevoxel, has been historically used. This means, every voxel is measuredand tested for an existence of the abnormality. As being the smallestunit of imaging, the voxel-based analysis carries the maximum amount ofanatomical information and in theory it is capable of detecting any typeof abnormalities; thus evaluation with lower granularity levels are notnecessary. This type of analysis, however, fails to replace humanjudgment; a human does not evaluate images in voxel levels. Voxel-basedanalysis is widely used for quantitative analysis of brain Mill. Whileit provides the highest granularity level of spatial information (i.e.each voxel), the sheer number of the voxels and noisy information fromeach voxel often leads to low sensitivity for abnormality detection.Thus, the primary reason of the failure is that information from eachvoxel is noisy and there are too many voxels.

To ameliorate this problem, spatial filtering, which effectively makesthe voxel size larger, has been used, leading to decreased granularitylevels. However, as granularity is reduced, information may also belost. As another means of ameliorating this issue, it is common tointroduce granularity reduction by applying isotropic spatial filtering.However, again, this type of isotropic reduction of the imagegranularity level is not what human does; they control granularity basedon anatomy.

An embodiment of the invention builds upon PCT/US14/69161 that disclosesthat the image granularity level can to be controlled based on anatomyand this describes how it can be achieved.

In this disclosure, a structure definition file is used. This typicallyis a structure-template atlas which contains pre-defined structures. Foranatomical reference and quantitative analysis of medical imaging data,atlases play crucial roles. Inside the atlases, locations and boundariesof various structures are defined. This knowledge is then applied to themedical image of interest. For example, given an atlas of the humanbody, in which 1,000 structures are defined, by applying this knowledgeto a whole-body CT image, these 1,000 structures can be defined manuallyor with computational aid such as warping the atlas to the image. Oncethe 1,000 structures are defined, then their volumes can be measured tocharacterize the anatomical features of the patient. In some embodimentsof the invention, the term “granularity” is used to describe the levelof fineness of defined structures in the atlas. If 10,000 structures aredefined, instead of 1,000, the anatomy of the body can be characterizedin more detail. Atlases are usually created as a generic purpose andthere is often no a priori assumption about how it will be used forwhich pathological states. As such, it is difficult to pre-determine themost appropriate granularity level of the atlas. In the medical image,the finest granularity is determined by the imaging voxel. The lowestgranularity level is the entire object of interest. There are almostinfinite numbers of granularity levels to choose and super-structures tocreate. The proper creation of the hierarchical tables to control thegranularity levels and the generation of various super-structures thatconsists of multiple voxels would be a very powerful method to fullyexploit the anatomical information encoded in medical images. Thus,depending on the medical or biological questions, a cellular analysismay not always be better than an organ-level analysis to answer amedical question.

FIG. 1 depicts a template atlas that defines 200 structures. Forexample, this figure shows in a first step of one embodiment that abrain image is parcellated into approximately 200 defined structures.

In the second step of one embodiment, this structural template isapplied to another image of a patient of interest, automaticallydefining the 200 structures in the patient. This can be also achieved bypreparing multiple template atlases, which are all applied to thepatient image and by performing a population-based labeling of the 200structures.

In the third step of one embodiment, the first and second steps can berepeated to various normal and abnormal brains. From the abnormalbrains, we can calculate aggregated reports such as the average volumeand image intensity of each defined structure. Based on these normalvalues, we can judge if the values from a patient is statisticallyabnormal or not. For example, if we are measuring volumes of the 200defined structures, we can obtain 200 statistical results, detectingabnormally large or small structures out of the 200 defined structures.

FIG. 2 depicts automated segmentation of the brain into 200 structuresby applying the template atlas information. This converts the imageinformation into a 200-element vector, representing the volumes of alldefined structures.

For example, in this example, a patient brain is parcellated to the 200structures and the volume of each structure can be measured. The valuescan then be compared to normal values and the abnormally smallstructures can be highlighted (values three standard deviations offnormal values are indicated by light blue and four standard deviationsoff are by dark blue).

In the fourth step of one embodiment, the granularity levels can bedynamically controlled by using a hierarchical relationship table. Oneof the most natural ways to create a hierarchical relationship is basedon the structural criteria following the development. For example, thebrain can be divided into different numbers of structures as shown below

FIG. 3 depicts a hierarchical definition of the brain structures withdifferent levels of granularity.

In the lowest granularity (Level 1), the brain is defined as one entityand the volume measurement yields the whole brain volume. In the nextlevel of the granularity, the brain is separated into five differentregions; the total brain volume is divided into five volumes, whichgives the information about the basic proportion of the brain. Forexample, ataxia patients would have disproportionally smallmetencephalon (cerebellum). Likewise, the brain can be divided intosmaller and smaller units as we go down the levels of the hierarchicaltree. In this example, the lowest level (Level 5) contains theinformation about the volumes of the 200 structures defined in FIGS. 1and 2.

An important aspect is that while the information from the lowest level(highest granularity) has the more amount of information (for example,the largest number of defined structures, say, 200), the information atthe higher level may not be obtained from the lower level information.

If we keep increasing the granularity, the highest granularity we canachieve is one voxel. For example, if we have 1×1×1 mm voxel resolution,one brain with 1.2 L of volume would have 1.2 million voxels. Each voxelcould serve as one structural unit. One assumption we can make is, if weexamine every single voxel, the entire brain can be examined, andtherefore lower granularity analyses, in which multiple voxels areinevitably combined, is not necessary. This assumption is not alwaystrue because in reality, identification of corresponding voxels acrosssubject contains inaccuracy and information from each voxel is noisy.This leads to the necessity to group and average voxel properties, whichwould increase the signal-to-noise. This operation is typically done byapplying isotropic spatial filtering, meaning voxels within a predefinedradius are averaged, effectively reducing the image spatial resolution.

This disclosure makes clear that, when we group voxels, we shouldprovide anatomical knowledge and voxels should be grouped based onanatomy. Our Level 5 atlas provides anatomical knowledge to definestructures. As we decrease the granularity (going up the Levels),multiple structures are combined, which provides different views toexamine the brain anatomy. For example, the “frontal lobe” defined inLevel 4 may provide 5% loss of volumes. This may not be detectable bymeasuring the volumes of the five constituents of the frontal lobe inLevel 5 because 5% loss is too small to detect in Level 5 due to anexpected increase in the noise in the lower levels. Conversely, theLevel 5 analysis can find that the 5% loss of the frontal lobe is due to20% volume loss of only one of the five constituents, say, superiorfrontal gyms, which can provide a more specific view than saying 5%volume loss of the frontal lobe. In this manner, analysis at differentgranularity levels can provide different anatomical views about theanatomical status, compared to the single-level analysis.

Thus, FIGS. 2 and 3 show a computer-implemented method, computer systemor computer readable medium for segmenting a region of interest 102 of asubject. The method can include receiving imaging data that can includea plurality of image elements of the region of interest 102 of thesubject. The method can also include segmenting, using at least one dataprocessor, the imaging data of the region of interest of the subjectinto a plurality of sub-regions corresponding to various structures 104at a plurality of levels of granularity, the plurality of levels ofgranularity can have a hierarchical relationship such that a level ofgranularity has fewer structures than a lower level of granularity. Themethod can further include calculating at each of the plurality oflevels of granularity an abnormality risk factor for the segmentedstructures of the region of interest 102. In one embodiment, thestructural units can be obtained in a variety of ways, each of the unitsbeing defined in a particular way. Higher-order structures can bedefined by combining the parts, and multiple ways of defining thehierarchy are possible.

In another embodiment, the method can further include providing at leastone template atlas for the region of interest of the subject. In thisembodiment, the various structures at each level of granularity can bepredefined in the at least one template atlas, and the segmenting caninclude applying the at least one template atlas to the received imagingdata at each level of granularity. In one embodiment, the receivedimaging data can be co-registered to the at least one template atlas. Inthis embodiment, the structural units can be pre-defined in the atlases,which are transferred to the imaging data. The scheme of thehierarchical relationship in one embodiment should be compatible withthe pre-defined structures in the atlases; if the hierarchicalrelationship is based on structural units A, B, C, D, these structuresneed to be defined in the atlases.

In some embodiments, the plurality of sub-regions corresponding to thevarious structures are non-randomly selected, and they can be grouped bya same matter or tissue (e.g., gray matter, white matter). For example,anatomical features can be used as a reason to combine the variousstructures into groups. Thus, the groupings can obey a biological system(e.g., central nervous system).

In one embodiment, the at least one template atlas can include at leastone of normal region of interest template data and abnormal region ofinterest template data. In one embodiment, for each atlas in the atleast one template atlas, the method can include filtering the image toa granularity of the atlas before co-registering the image to the atlas.

Further, the segmenting can include applying the received imaging datato the normal region of interest template data and/or abnormal region ofinterest template data. Further, the calculating can be based on thereceived imaging data fitting within either the normal region ofinterest template data or the abnormal region of interest template dataat a statistically significant level.

Additionally, the segmenting can include measuring at least one ofsizes, shapes and intensities of the various structures at each level ofgranularity, and performing population-based labeling of the variousstructures to characterize anatomical features of the region ofinterest. For example, an anatomical feature can be a characteristic ina leg such as a tissue size. Suppose a researcher is interested inanatomical features that can predict the risk of a walking disabilitydue to broken bones in aged populations. After finding 200 clinical datapoints in which 40 patients suffered from the walking disability, theresearcher can measure the volumes of all 1,000 defined structures ofthe 40 patients and compare them to the 160 subjects without walkingdisability. He can find that the volumes of the several bones in thelegs and the feet were significantly lower.

In this hypothetical scenario, there is no guarantee that the atlasemployed in the study defines the structure in the way most appropriatefor the study. First of all, there are almost infinite ways to definestructures in an atlas. The entire right leg bones can be defined as a“leg bone” or the four major bones in the leg can be defined separately.Then one part of the leg bone, say the tibia, can be divided furtherinto smaller units such as the lateral tibia condyle, the medial tibiacondyle, etc. Such sub-divisions can be extended further intomicroscopic levels such as intra-bone structures and constituent cells.The more structures are defined in the atlas, the more anatomicalinformation is added. Going back to the example of the risk predictionfor the walking disability, the volume of the entire leg bone could be agood marker for the prediction. It is also possible that the loss ofbone is a systemic event and the bone mass of the entire body could beeven a better marker. We cannot also deny a possibility that the “legvolume”, which is the combination of the leg bones and leg muscles,would be a better marker, because the amount of the muscle could also bean important factor for the prediction. Embodiments of the inventionpoints out that anatomical entities (called “super-structure”) atdifferent granularity levels can be generated by combining smalleranatomical units and the way super-structures are generated can beguided by hierarchical tables based on anatomical knowledge. In oneembodiment, the anatomical feature can be at least one of anabnormality, a disease, a condition, a diagnosis, or any combinationthereof.

In one embodiment, the measuring can include measuring at least one of:a mean intensity of the image elements in the sub-regions; a sum ofintensities of the image elements in the sub-regions; a highestintensity of the image elements in the sub-regions; and a lowestintensity of the image elements in the sub-regions.

In one embodiment, the method can further include providing othersegmented imaging data having other structures of regions of interest ofother subjects; and identifying a substantially same region of interestin the other imaging data as the region of interest of the subject bycomparing one or more anatomical features, at multiple levels ofgranularity, in at least one of sizes, shapes and intensities of thestructures of the segmented imaging data and the other structures of theother segmented imaging data based on similarities of the structures andthe other structures.

In one embodiment, the identifying step can use a single feature vectorincluding measures of intensity of the image elements in thesub-regions. The identifying step can include comparing the featurevector to one or more feature vectors of the other imaging data. In oneembodiment, the identifying can include the feature vector exceeding apredetermined threshold. In another embodiment, the identifying furtherincludes incorporating at least one non-anatomical feature in themapping.

In on embodiment, the at least one non-anatomical feature can include atleast one of diagnosis, functions or other clinical information.Previously, similarity measures or clinical correlations were based onthe features of the 1 million voxels. However, 1 million voxels ofinformation is noisy and includes too much data, so many of them are notrelevant for important features. Thus, the hierarchicalmulti-granularity analysis turns the 1 million voxels into structures,upon which the subsequent analysis such as searching and correlation canbe performed.

In one embodiment, the method can further include dynamicallycontrolling the plurality of levels of granularity using a hierarchicalrelationship table.

In another embodiment, a structure of a template atlas at a level of thehierarchy can include structures of a template atlas at a lower level inthe hierarchy. Further, an area of the image can be mapped to differentsub-regions corresponding to different structures of different atlases.

The imaging data can be generated from at least one of magneticresonance imaging (MRI), computed tomography (CT), positron emissiontomography (PET), ultrasound, or nuclear tracer three-dimensionalimaging. Further, the region of interest can include at least a portionof at least one of a brain, a heart, a liver, skin, a lung, anotherorgan, one or more bones, or any combination thereof.

Another important point is that the lower level (high-granularity) datais difficult to connect to human perception. When humans see anatomy,they see anatomical objects from a high level point of view. Forexample, when a doctor sees the brain of a patient, he first sees thestatus of the entire brain and checks the level of the atrophy. Thens/he examines the gray and white matter in the five lobes. Quantitativeanalysis at this level of granularity is important to connect thelow-level (high granularity) information to human's perception. At thehigher level, it is more straightforward for doctors to appreciate thequantitative results and connect them to his perception. As he moves tolower levels, the doctor can gradually rely on more of the quantitativedata because it becomes difficult to visually or conceptually grasp thequantitative results. The voxel-level (lowest level and highestgranularity) results cannot often be visually confirmed because doctorsdo not see images in the voxel level. For an image-analysis product tobe accepted in the market, the hierarchical multi-granularity dataanalysis and presentation can be important.

The following discussion of a study illustrates these advantages. InDjamanikova et al., “Tools for multiple granularity analysis of brainMill data for individualized image analysis,” Nueroimage. 2014 Nov. 1;101:168-76, we tested our multiple-granularity analysis using eightAlzheimer's disease (AD) patients and ten patients in an age-matchedcontrol. We divided the brains into 5 granularity levels, defining 11,17, 36, 54 and 254 structures in the brain. With this small number ofpatients, we did not find structures that were statistically differentbetween the AD patients and the control subjects when the brains weredivided into 36, 54, and 254 structures (high granularity analysis), butstructures, such as the telencephalon, defined in the 11 and 17 labels(lower granularity analysis) could detect statistically significantvolume losses in the patient group. This exemplifies how we can optimizethe structural granularity levels to detect biological events ofinterest.

This study proposes a systematic reduction of the spatial informationbased on ontology-based hierarchical structural relationships. Forexample, 254 brain structures were first defined in multiple (n=29)geriatric atlases and five levels of ontological relationships wereestablished, which further reduced the spatial dimension as few as 11structures. The multiple atlases were then applied to T1-weighted MRI ofeach subject data for automated brain parcellation. Then at eachontology level, the amount of atrophy was evaluated, providing a uniqueview of low-granularity analysis. This reduction of spatial informationallowed us to investigate anatomical phenotypes of each patient, whichwere demonstrated for Alzheimer's disease and primary progressiveaphasia in patients.

To analyze images from multiple subjects, identifying anatomicallycorresponding locations across the subjects is one of the first steps. Awidely used approach is to define specific target structures, such asthe hippocampus, manually, which is called the region-of-interest (ROI)approach and is considered the gold standard for the neuroanatomicalresearch. This approach is, however, applicable to only a small portionof the anatomical structures. For example, with a 1 mm isotropic spatialresolution, a brain with a 1.2 L volume would have 1.2 million voxels.The hippocampus volume is typically about 4,000 voxels (4 ml), meaningonly 0.3% of the entire voxels are evaluated. Voxel-based analysis is analternative analysis, in which correspondence of the entire 1.2 millionvoxels between two brains are established automatically (see e.g. [1]).Suppose we have 50 control and 50 patient images, the entire dataset canbe expressed as two matrices of [(50 subjects)×(1.2 millionvoxels)]Control, Patient. This voxel-vector of (1.2 million voxels)needs to be re-ordered, such that any arbitrary vector element, say, ithvoxel of the 1.2 million-element vector, identifies the same anatomicallocations across the 100 subjects. Then, we can contract the 50-elementpopulation dimension to the average and the standard deviations; the twomatrices are now [(average, standard deviation)×(1.2 million voxels)].The actual measurements can be voxel intensity (e.g., T2, fractionalanisotropy, mean diffusivity) or morphometric parameters such asJacobian, representing local atrophy or hypertrophy. This contractioncan enable us to perform t-test at each voxel independently, identifyingvoxels with significantly different values between the two populations.

The voxel-based analysis is powerful because it retains the maximumamount of location information until the final statistical analysis; theentire brain is examined at the highest-possible granularity level,i.e., 1.2 million voxels. However, the limitation of this approach isalso widely recognized (see e.g. [2]). First of all, the informationeach voxel carries is noisy. This issue is magnified by that fact thatthere are 1.2 million intricately dependent observations. Second, theaccuracy of voxel-based registration is often in question (the 1.2million voxel-vectors may not be well aligned across subjects). This isespecially the case for two reasons; 1) lack of contrast: thevoxel-to-voxel mapping between two corresponding regions is not accurateif the regions lack contrasts and 2) anatomical heterogeneity: excessiveanatomical variability in certain areas, such as cortical folding, couldprevent us from accurately identifying corresponding voxels between twobrains in such areas. To ameliorate these issues of the high-granularityobservation, it is common to reduce the level of granularity by applyinga spatial filter, effectively reducing the image resolution throughvoxel-averaging (FIG. 4). FIG. 4 depicts comparison of granularityreduction by isotropic resolution reduction (upper row) andontology-based structural reduction (bottom row).

In the study, we provide tools using an alternative approach to analyzethe 50×1,200,000 matrices based on two concepts. First, in many clinicalstudies, even if the patient population is as homogenized as possible bystringent clinical criteria, a considerable amount of anatomical and,potentially pathological heterogeneity remains. After all, if theclinical information encodes enough information to purely define thepatients with the same pathology, there would be less need for imagingstudies; namely clinical information should be sufficient to describethe pathology. A primary interest is, therefore, to characterize theanatomical heterogeneity within a patient group. Namely, differentpatients may have abnormalities in different locations. If so, ourinterest is the first subject dimension (e.g. n=50) of the matrices andthe group-aggregated statistics (reduction of the average and standarddeviation) at each location is no longer an appropriate analysis. Thisleads us to an alternative concept, which is the reduction of the secondlocation dimension (n=1,200,000). In VBA, this is achieved by spatialfiltering. While this remains an effective approach, even with 83reduction of the voxel size, the level of granularity remains high(2,300) while considerable amount of anatomical information is lost. Forfurther reduction of the location dimension, anatomy-specific filteringseems a logical approach, in which a large number of voxels are groupedbased on pre-defined anatomical criteria, called atlases.

This anatomy-specific filtering based on a pre-defined atlas, however,has several issues. First, the number of defined structures is limitedby available image contrasts. T1-based contrast could define up toseveral hundred structures. If there are 300 defined structures, eachstructure has 4,000 voxels in average. Compared to VBA, the level ofgranularity is substantially low, potentially making the measurementinsensitive to highly localized abnormalities. Second, there aremultiple criteria to define structures and, depending on pathology,different criteria may be used. For example, for vasculature diseases,brain parcellation based on the vasculature territories may make moresense than classical ontology-based brain parcellation. Third, theaccuracy issues of VBA due to the lack of the contrasts andcross-subject variability still exist for the structure-based analysis,although they may influence the results in different ways. Once thevoxels are grouped to define a structure, the location information ofeach voxel inside the structure can degenerate and there is no longer avoxel-wise accuracy issue. Instead it may manifest as the accuracy ofthe boundary definition.

In this study, we developed a tool that can flexibly change thegranularity level based on hierarchical relationships of 254 structuresdefined in our atlas. We tested this tool within a framework of amultiple-atlas brain parcellation algorithm [3-9]. Using 29pre-parcellated atlases, test data were automatically parcellated intothe smallest structural units (254 structures). Then, these structureswere dynamically combined at five different hierarchical levels, down to11 structures [10, 11]. This tool was first applied to a control groupto measure test-retest reproducibility and the normal range ofanatomical variability. Then we analyzed two groups of dementiapopulations for demonstration: Alzheimer's disease (AD) and primaryprogressive aphasia (PPA).

Methods:

Subjects

Three study groups were used for this study: young adult and elderlycontrols, AD patients, and PPA patients. All studies were approved bythe Institutional Review Board of Johns Hopkins University and writteninformed consent was obtained.

Young adult subjects: The database for normal adult subjects wasobtained from previous studies (n=17, age mean=31 years old, age range22 to 49 years old) [12], in which each subject was scanned twice twoweeks apart. Scan parameters are MPRAGE, matrix 256×256, FOV 256 mm×256mm, slice thickness 1.2 mm, TE 3.15 ms, and TR 6.747 ms. These data wereused to measure test-retest precision of the method and anatomicalvariability within the young normal subjects.

Alzheimer's disease (AD) patients and elderly controls: We used AD andelderly data from a study of a well-characterized group of individualsconducted by the Johns Hopkins Alzheimer's Disease Research Center(ADRC), with written informed consent in accordance with therequirements of the Johns Hopkins Institutional Review Board and theguidelines endorsed by the Alzheimer's Association [13]. Detaileddemographics, health, clinical features, and initial findings werereported previously [14]. Briefly, the study sample comprised 8 patients(mean age, 75.6) who met NINCDS/ADRDA criteria for AD [15] and had aClinical Dementia Rating (CDR) of 1 and 10 individuals (mean age, 74.3)who were cognitively normal and had a CDR=0 (normal controls or NC). Thedemographic characteristics of the subjects were as follows: AD—meanage=75.6 years, mean education=15.7 years, male/female=5/3 and NC—meanage=74.3, mean education=16.2 years, male/female=3/7. Subjects wereexcluded from enrollment if they were under the age of 55, had a historyof a neurological disease other than AD, or a history of majorpsychiatric illness. As previously described [14], there were nodifferences among these groups with regard to age, sex, race, education,and the occurrence of vascular conditions, such as hypertension,hypercholesterolemia, and heart attack. Written, informed consent wasobtained under the oversight of the Johns Hopkins Institutional ReviewBoard using guidelines of the Alzheimer's Association [13]. MPRAGE scanwas conducted according to the protocol of the Alzheimer's DiseaseNeuroimaging Initiative (ADNI) [16], with an echo time of 3.2 ms and arepetition time of 6.9 ms. The imaging matrix was 256×256, with a fieldof view of 240×240 mm, zero-filled to 256×256 mm and a sagittal slicethickness of 1.2 mm

Primary progressive aphasia (PPA) subjects: n=6, mean age=70 years old,age range 56 to 84 years. Scan parameters are MPRAGE, matrix 256×256,FOV 230 mm×230 mm, slice thickness 1 mm, axial, TE 6 ms, and TR 10 msand MPRAGE, matrix 256×256, FOV 212 mm×212 mm, 1.1 mm slice thickness,axial, TR 8.436 ms, TE 3.9 ms. The participants with PPA were seen inone of the author's (A.H.) outpatient cognitive neurology clinic at theJohns Hopkins Hospital and agreed to participate with written informedconsent in accordance with the requirements of the Johns HopkinsInstitutional Review Board. They were diagnosed with PPA on the basis ofhaving a predominant and progressive deterioration in language in theabsence of major change in personality, behavior or cognition other thanpraxis for at least two years (Mesulam, M. M. (1982). Slowly progressiveaphasia without generalized dementia Annals of Neurology 11, 592-598).

Atlas Inventory:

In this study, we used multiple atlases (JHU T1 Geriatric Multi-AtlasInventory) to perform automated brain parcellation. This atlas inventoryis designed for geriatric patient populations with potential brainatrophy. The data were based on a portion of the AD/Elderly populationdescribed above; AD patients (n=15, age mean=73 years old, age range 56to 80 years old) and normal elderly controls (n=14, age mean=75 yearsold, age range 60 to 80 years old). These images were parcellated into254 structures defined in our JHU brain atlas (Eve atlas) [17-19]. Thissingle-subject atlas was initially warped to the 29 multiple atlasesusing a method described by Djamanakov et al [20], followed by manualcorrections for mislabels.

Image Processing

The multiple-atlas brain parcellation followed the following steps:

All T1-WIs were bias corrected and skull-stripped using SPMS (TheWellcome Dept.

of Imaging Neuroscience, London; www.fil.ion.ucl.ac.uk/spm). Afterinitial linear alignment, all atlases were warped to the subject imageusing Large Deformation Diffeomorphic Metric Mapping (LDDMM) [19, 21,22]. The transformation matrix was then applied to the co-registeredparcellation maps of each atlas. The details of the multi-atlas fusionalgorithm used in this study are described in our previous publication[23]. Briefly, let A be a set of atlases, with manual labels A=(I, W),where I denotes the gray-scaled T1-WI and W denotes the manualsegmentations of I. Given a to-be-segmented subject with imageintensity, I of the subject, at voxel i modeled as conditional Gaussianrandom field, conditioned on the unknown atlas and the correspondingunknown diffeomorphism. The algorithm for segmentation iterates atlasselection and diffeomorphism construction as a variant of theexpectation-maximization method. Define the Q-function as theconditional expectation of the complete-data log-likelihood accordingto:

Q(W;W ^(old))=E{log p(I,W|A)|I,W ^(old)}=Σ_(i)Σ_(a) P _(A) _(i) (a|I,W^(old))log p(I,W|a).

where the sum is obtained over all voxels i and atlases a. Then thesequence of iterates W(1), W(2), . . . , is defined by the iteration:

$W^{new} = {\underset{W}{\arg \; \max}{Q\left( {W;W^{old}} \right)}}$

whose calculation is alternated with the calculations of the conditionalprobabilities) P_(A) _(i) (a|I,W^(old)). P_(A) _(i) (a|I,W^(old)) isderived from the conditional mean of the indicator function and encodesthe set of atlases being selected in the interpretation (Tang et al.,2013).

Ontology-based multi-granularity analysis using RoiEditor:MriStudio

Analyses were performed using the final parcellations I the native spaceof the subjects. The 254 structures defined in the parcellation map wereassigned a hierarchical relationship based on their ontologicalrelationship as tabulated in Table 1. This relationship consists of fivehierarchical levels. As the level goes up, the granularity of structuraldefinition increases as; 11-17-36-54-254. This relationship isimplemented in RoiEditor (X. Li, H. Jiang, and S. Mori, Johns HopkinsUniversity, www.mristudio.org) as shown in FIGS. 4 and 5. FIG. 5 depictsa screenshot of the RoiEditor interface that allows automatedvisualization and quantification of the ontology-based multi-granularityimage analysis. As the brain is parcellated to the multi-levelstructures, the sizes of all structures are calculated automaticallyusing this software. It is important to note that only one parcellationwas done for each subject, at the highest granularity level. Thesubsequent reparcellation to different granularity levels was achievedby the recombination of individual ROIs to create new larger ROIs asdefined by the various levels of granularity. The hierarchicalrelationship can also be user-defined through the text file.

Test-Retest Measurements of Multi-Atlas Segmentation:

To test the test-retest reproducibility of the whole-brain multi-atlasparcellation method, the data from the twice scanned young adultsubjects (n=17) were utilized. The volume data was compared for allregions of each subject across the two scans. From this dataset, thetest-retest reproducibility for each subject was measured. In addition,anatomical variability among the 17 normal subjects was measured. Thetest-retest measurement precision and the anatomical variability werethen compared using a principal component analysis.

Characterization of Anatomical Phenotypes of AD and PPA Patients:

To characterize anatomical phenotypes of AD and PPA patients,multi-atlas segmentation was performed on all patients and themulti-granularity-level analysis was performed. The anatomical featureof each patient was presented by z-scores based on the age-matchedcontrol data.

4.3 Results:

Test-Retest Reproducibility:

The test-retest variability (percent volume difference) across all ROIsbetween the two scans was found to be 2.6%±1.7%, 1.7%±1%, 1.4%±0.7%,1.5%±0.7%, 1.5%±0.9% for the five different granularity levels. FIG. 5Bis a plot of test-retest analysis using 17 healthy subjects, scannedtwice. The variability between scans 1 and 2 is plotted as a percent ofROI size for each ROI measured at three highest granularity levels. Thex-axis is the size of the structures in voxels (log 10). FIG. 5B showsthe relationship between the variability and the size of theparcellation at three different granularity levels (Levels 3, 4, 5). Aclear inverse relation can be seen, in which the variability increasesdrastically for structures less than 1000 voxels, while most of thestructures larger than 1000 voxels have a small amount of variability(<2%). At Level 4, there are only two structures that are less than 1000voxels in size and none in Levels 1-3. Consequently, the improvement inthe average test-retest variability for Levels 1-4 was negligible.

FIG. 6 shows results of principal component analysis (PCA) using the 254ROIs in the highest granularity level. The plot based on the first threeprincipal components clearly isolates anatomical features of the 17normal subjects with respect to the test-retest variability, suggestingthe test-retest precision of this approach is high enough tocharacterize anatomical phenotypes of the normal population.

Anatomical Variation Among the Normal Subjects:

FIGS. 7A and 7B show the anatomical variability at two differentgranularity levels (level 1 and 4) of segmentation for the young normaladults. A few things to note are: 1) the lateral ventricles are the mostvariable features within normal adult populations, given that theaverage level of variability is at ˜40% and 2) as level of granularityincreases (and thus each defined structure becomes smaller), thevariability tends to increase, which could be the combination of trueanatomical variability and reduction of parcellation accuracy. Forexample, in level 1, the left telencephalon shows variability of 2.6% inthe normal population. At level 2, regions that comprise the lefttelencephalon are the left cerebral cortex, the cortical nuclei, and thewhite matter (see Table 1). Their average variability was 4.0%. Furtherbreaking down these regions into smaller subregions (i.e. the cortex isdivided into the frontal, parietal, temporal, limbic, and occipitalcortices) at level 3, the average variability at this level is 5.8%.This indicates the general trade-off between finer localizationinformation and measurement precision. We can expect that the highergranularity levels, in general, provide more information about the localshape variability. For example, the Level 4 data suggest that the largepopulation variability of the ventricle seen in Level 1 is mostly due tothe large variability of the anterior and posterior lateral ventricles,while the third and fourth ventricles have much less variability.

Comparison of AD and Age-Matched Control Groups

FIG. 8 shows the “classical” view of anatomical abnormalities of the ADpopulation. The graphs show that, at the lowest granularity level, astatistical difference was found at the ventricles (hypertrophy) andtelencephalon and diencephalon (atrophy) between the AD and NCpopulations. As the granularity level increases, a more detailed view ofthe tissue atrophy can be obtained. At level 4, 20 structures reachedstatistical significance (p<0.05) between the two groups. However, aftera Bonferroni multiple comparison correction, none of the regions fromlevels 3-5 were significantly different (p<0.05).

Individual Views of Anatomical Phenotype

FIG. 9 shows an alternative view of the same data at level 5, in whichthe within-group data reduction was not performed and the anatomicalphenotype of each individual was delineated using z-scores. In thisanalysis, we first calculated the average and standard deviations of thevolume of each structure from the age-matched controls first and thencalculate the z-score. None of the structures in the control groupreached z-score higher or lower than 2. On the other hand, many“relatively” atrophic (indicated by pink) and hypertrophic (indicated bygreen) structures exist in the AD group. For example, the ventriclesstand out as regions where many AD patients deviate substantially fromthe mean. However, as shown in the two example cases shown in FIG. 6,the anatomical variability within the AD population is striking. Evenwith this AD population with stringent inclusion criteria, thestructure-by-structure population averaging could lead to: 1) loss ofimportant patient specific anatomical features and 2) lower sensitivitydue to inclusion of patients with abnormalities at different anatomicallocations.

Compared to the AD population, we expect even further within-groupanatomical variability in the PPA population; these patients have beendiagnosed under the umbrella term, primary progressive aphasia, based ontheir cognitive symptoms, specifically language memory deficits, whichis known to contain various pathological mechanisms. FIG. 10 showsimage-based representation of the multi-granularity analysis of one PPApatients. In this patient, the level 1 granularity analysis revealsrelative size difference between the two hemisphere (the left hemisphereis smaller). As the level 2 analysis indicates lobar-level anatomicalfeatures, indicating temporal atrophy. The highest granularity level(level 5) indicates global atrophy of the left cortex and associatedwhite matter regions, while the temporal lobe has the most severeatrophy. FIG. 11 shows image-based presentation of three representativePPA cases using T1-superimposition views at level 5, in which thecolor-coded z-score information is superimposed on their T1 weightedimages. At a glance, we can appreciate that Patient #1 has globalcortical atrophy in both hemisphere. In Patient #2, atrophy is highlyfocal at the left temporal lobe.

4.4 Discussion:

Tradeoff Between Granularity and Variability

The measurements of test-retest reproducibility of our automated brainparcellation by a multi-atlas approach indicated that the measurementprecision was high with respect to anatomical variability among thenormal subjects (FIG. 6). The test-retest precision becomes lower as thegranularity increases, and at the highest granularity level, thereproducibility was 2.6+/−1.7% for all 254 measured structures. Therecan also be a tendency for the amount of anatomical variability amongthe young normal subjects to increase as the granularity increases (FIG.7). This is probably due to the mixture of real anatomical variabilityand decreased level of measurement precision. In PCA (FIG. 5B), wecalculate total variance of two measurements of the 17 subjects. Thisvariance contains scanning reproducibility (test-retest), automatedsegmentation errors, cross-subject variability, and other sources ofvariability. PCA attempts to find sources of variability in themeasurements and determines three most dominant sources, which, in thiscase, accounts for 65.9% of the total variance. The three axes are thecombination of measured structures and thus do not have immediateanatomical meaning. What is important is that the two repeatedmeasurements were naturally clustered in the PCA space with respect tothe cross-subject variability. Therefore, we can conclude thatreproducibility of the automated quantification method is high comparedto expected amount of cross-subject variability.

The multi-granularity analysis was also applied to a well-characterizedAD population and an age-matched control group. Our study sample sizewas small to draw a solid conclusion about anatomical abnormalities inthis AD cohort but there are several findings. First, from thetest-retest reproducibility and the anatomical variability among thenormal population, the statistical power to detect the group differencediminishes as the granularity increases. The lower granularity analysiscould detect statistical differences after multiple-comparisoncorrection, but the findings lack detailed biological insights into theAD pathology; for example, the level 1 analysis simply tells us the ADpopulation has brain tissue atrophy and enlarged ventricles. On theother hand, if hypothesis-driven measurements of, for example,hippocampal volumes of an AD population suggest 10% volume loss and,simultaneously, our low-granularity analysis suggests 10% volume loss ofthe entire gray matter, the conclusion that “hippocampal volume isdifferent in AD population compared to the control population” may bemisleading because it singles out one structure as opposed to the entiregray matter. Thus it is important to analyze not only structures ofinterest, but also their substructures, and greater areas to which thestructures belong. From the same T1 data sets, different granularitylevels offer multiple options to analyze the data with differentstatistical power and the different amount of anatomical specificity.From the specificity point of view, one could argue the sensitivity ofanalysis is at the highest when the granularity level matches thespatial extent of the abnormality; we may lose the sensitivity when thedefined structures are too large (thus includes non-affected regions) ortoo small (divides the atrophic areas into too many regions).

Brain Parcellation Criteria

The above argument may lead to a fundamental question to allparcellation-based image analysis; “are we parcellating the brain withproper anatomical criteria?” For example, we know that the distributionof ischemic areas follow vasculature territories, but not the tissuetype. If our interest is to find the affected vasculatures, the atlas weemployed, which is based on tissue types, may not be appropriate. On theother hand, if we want to identify brain structures and associatedfunctions which are affected by an infarction, a parcellation schemethat represents brain functional distribution would be needed.

The multi-granularity parcellation scheme we offer is based on the brainontology used by the atlas by Mai et al as well as Allen BrainInstitute. Here our assumption is the evolutionarily-conservedontology-based anatomical definition is one of the most suitable ways torepresent brain anatomy and functions. However, there exist multipleontology definitions in the brain, and thus our scheme cannot beconsidered as the gold standard. As a matter of fact, our ontology isoften not compatible with structural definitions with which we are mostfamiliar. For example, in radiological descriptions, we often define thebrain constituents as the cortex, the white matter, the deep graynuclei, the brainstem, and the cerebellum. The brainstem is furtherdivided into the midbrain, the pons, and the medulla. However, classicalontology divides it into the mesencephalon, metencephalon, andmyelencephalon, in which the mesencephalon includes the pons and thecerebellum together. In many ataxia patients, atrophy often occurs inthe pons and the cerebellum together and for a low granularity analysis,the classical ontology-based analysis could be more appropriate, butthere is certainly a large degree of freedom in defining hierarchicalrelationship of the brain structures.

There are several important issues related to this topic. First of all,when we define ontology-based atlases, the criteria to parcellate thebrain and the way we define hierarchical relationships are two differentissues. The former could lead to multiple structural definitions whichare mutually exclusive. For example, the same brain could be parcellatedbased on tissue type, classical brain structural definitions,vasculature territories, cytoarchitectures, distribution of specificreceptors, etc. The latter is a question of how to combine structuresdefined in the highest granularity levels (in our case, 254 structures)and establish a hierarchical relationship. For the latter issue,RoiEditor provides a flexible interface to incorporate user-definedhierarchical relationships through the ontology table shown in Table 1.

The latter is also possible if users have their own brain parcellationmaps. However, while the latter issue (how to combine structures andbuild an ontology relationship) is purely an issue of image analysisAFTER the image parcellation at the highest granularity is complete, theformer issue is related to how we should parcellate the image to beginwith. This issue is discussed more in detail in the next section.

One interesting question is what defines the highest granularity level:Why does our parcellation map not contain more than 254 structures? Theparcellation criteria is not always exact science and arbitrary judgmentis involved. However, it is generally driven by available imagecontrast. For example, the whole hippocampus is defined as onestructure, even though we know that the hippocampus consists of manysubstructures. This is because we lack both resolution and contrast tosub-divide the hippocampus using conventional MRI of live humansubjects. As a matter of fact, because our parcellation map was createdbased on T1, T2, and DTI contrasts, there are certain structures thatare delineated in the atlas, but are invisible in T1-weighted images.For example, the pons area is divided into the middle cerebellarpeduncle, the corticospinal tract, and the medial lemniscus, which areclearly identifiable by DTI but not in T1. Therefore, T1-weightedimaging may not have ability to detect atrophy specific to thecorticospinal tract. While this level of structural granularity isrelevant for DTI, they may not be reliable for T1. Therefore, the usageand interpretation of the multi-level granularity analysis requiresanatomical knowledge. This issue is, however, not specific to ourmulti-level granularity analysis and can be applied to voxel-basedanalysis in general.

Multi-Atlas Approach

In this study, we employed our multi-atlas brain parcellation approach,called Diffeomorphic Probability Fusion (DPF), because we found it isgenerally more accurate than a single-atlas approach. However, bothapproaches operate under the same concept: 1) all voxels of the atlasesare mapped to the corresponding voxels in a patient based on imagetransformation, 2) the brain structures are defined in the atlases as“parcellation map”, and 3) the parcellation maps are then transferred tothe patient using the transformation results from step 1). If we haveonly one atlas, this is a single-atlas approach. If we have multipleatlases, multiple parcellation maps are cast to a patient brain and afusion process is required to combine the multiple maps. Ourontology-based analysis is independent to how the brain is parcellatedand can be combined to the both approaches.

In the previous section, the issue of brain parcellation criteria wasdiscussed. For example, we have four different types of the parcellationmaps in our single-subject atlas called “Eve Atlas”: the tissue type mapused in this study, vasculature map, resting-state functionalconnectivity map, and cytoarchitectonic map. If a single-atlas approachis used, application of these different brain parcellation criteria isstraightforward; we simply need to apply the transformation matrix toany parcellation maps of interest and warp the maps to the patient datareadily. This concept can also be easily applied to the multi-atlasapproach but, practically, preparing multiple atlases with multipleparcellation maps is a time consuming step that requires an extensiveamount of manual work. Currently we offer 29 geriatric atlases with the254 structural definitions. Other parcellation criteria such as thevasculature territories are not available at this moment.

Individualized Analysis

A big motivation to use the low-granularity analysis, as opposed to thevoxel-based analysis is that the large reduction of spatial informationfrom 1.2 million voxels to a mere 11-254 structures allows us toevaluate anatomical phenotypes of individual patients as shown in FIGS.9-11. In the high-granularity analysis such as voxel-based analysis, itis common to contract the population dimension and average all patientinformation for each individual voxel. This approach increases thestatistical power only when the entire population shares the abnormalityat similar locations. This is why homogenization of pathology within apatient population through stringent inclusion criteria is vital. Ourpreliminary analysis of the AD population, however, revealed highlyheterogeneous anatomical features (FIG. 9). The situation is far worsefor the PPA population, which is known to contain multiple pathologicalconditions (FIGS. 10 and 11). In these situations, we cannot considerMRI as a tool to conclude the pathological phenotype of the patientpopulation as a biomarker. Like many other clinical information, it isone of the weakly discriminating factors of the patient conditions. Ifthis is the case, the task of this ontology-based analysis is tocompress the 1.2 million spatial dimension into a much smaller andstandardized format, while, much like what jpeg file does tophotography, losing a minimum amount of pathology information. This isan important step, if we want to combine MRI-based anatomical featureswith non-image clinical information such as demography, life style,clinical symptoms, lab tests, etc., to improve our ability to stratifythe heterogeneous patient groups or predict the outcomes.

CONCLUSION

In this study, we introduced a new concept of low-granularity anatomicalanalysis based on ontology-based hierarchical relationships of the brainstructures. We combined this analysis with a multi-atlas parcellationapproach and applied to T1-weighted brain MRI for brain atrophyanalysis. Test-retest reproducibility was high. The anatomicalvariability of the normal population was measure at five differentgranularity levels, which could be used as an estimate of powercalculation. This approach was then applied to AD and PPA populations.The potential of this approach to perform individual-based anatomicalanalysis was discussed. The proposed approach was integrated intoRoiEditor for automated multi-granularity analyses.

It is important to note that the information captured at each level ofthe hierarchy may be represented using a single feature vector. Forexample, the information depicted in FIGS. 7A and 7B may be stored inthe same feature vector as information about levels 2, 3, and 5. Then,the feature vector, or any other data structure capable of storing suchinformation, may be compared with one or more other feature vectors (orany other data structure(s) capable of storing such information), todetermine whether the information is similar enough. Similarity may bemeasured in a variety of ways, such as by meeting certain confidenceintervals in statistical methods, etc.

Additionally or alternatively, the information captured at each level ofthe hierarchy may be represented using a single feature matrix. Forexample, the information depicted in FIGS. 7A and 7B may be stored inthe same feature matrix as information about levels 2, 3, and 5. Thematrix may be able to hold the same information as feature vectors, andmay hold a series of feature vectors, e.g. those representingmeasurements of a subject taken over different periods of time. Thus,the matrix may be able to represent progression of anatomicalcharacteristics over time. Just as feature vectors may be compared tofind other similar feature vectors, matrices may be compared to findother similar feature matrices, e.g. patients who have similar diseaseprogressions over time. Although these features are discussed in termsof matrices, it should be appreciated any other data structure capableof storing such information could be used, e.g. a feature vector.

A computing device may perform certain functions in response to aprocessor executing software instructions contained in acomputer-readable medium, such as a memory. In alternative embodiments,hardwired circuitry may be used in place of or in combination withsoftware instructions to implement features consistent with principlesof the disclosure. Thus, implementations consistent with principles ofthe disclosure are not limited to any specific combination of hardwarecircuitry and software.

Exemplary embodiments may be embodied in many different ways as asoftware component. For example, it may be a stand-alone softwarepackage, a combination of software packages, or it may be a softwarepackage incorporated as a “tool” in a larger software product. It may bedownloadable from a network, for example, a website, as a stand-aloneproduct or as an add-in package for installation in an existing softwareapplication. It may also be available as a client-server softwareapplication, or as a web-enabled software application. It may also beembodied as a software package installed on a hardware device.

Numerous specific details have been set forth to provide a thoroughunderstanding of the embodiments. It will be understood, however, thatthe embodiments may be practiced without these specific details. Inother instances, well-known operations, components and circuits have notbeen described in detail so as not to obscure the embodiments. It can beappreciated that the specific structural and functional details arerepresentative and do not necessarily limit the scope of theembodiments.

It is worthy to note that any reference to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. The appearances of the phrase “in oneembodiment” in the specification are not necessarily all referring tothe same embodiment.

Although some embodiments may be illustrated and described as comprisingexemplary functional components or modules performing variousoperations, it can be appreciated that such components or modules may beimplemented by one or more hardware components, software components,and/or combination thereof. The functional components and/or modules maybe implemented, for example, by logic (e.g., instructions, data, and/orcode) to be executed by a logic device (e.g., processor). Such logic maybe stored internally or externally to a logic device on one or moretypes of computer-readable storage media.

Some embodiments may comprise an article of manufacture. An article ofmanufacture may comprise a computer readable storage medium to storelogic. Examples of a computer readable storage medium may include one ormore types of computer-readable storage media capable of storingelectronic data, including volatile memory or non-volatile memory,removable or non-removable memory, erasable or non-erasable memory,writeable or re-writeable memory, and so forth. Examples of storagemedia include hard drives, disk drives, solid state drives, and anyother tangible or non-transitory storage media.

It also is to be appreciated that the described embodiments illustrateexemplary implementations, and that the functional components and/ormodules may be implemented in various other ways which are consistentwith the described embodiments. Furthermore, the operations performed bysuch components or modules may be combined and/or separated for a givenimplementation and may be performed by a greater number or fewer numberof components or modules.

Some of the figures may include a flow diagram. Although such figuresmay include a particular logic flow, it can be appreciated that thelogic flow merely provides an exemplary implementation of the generalfunctionality. Further, the logic flow does not necessarily have to beexecuted in the order presented unless otherwise indicated. In addition,the logic flow may be implemented by a hardware element, a softwareelement executed by a processor, or any combination thereof.

While various exemplary embodiments have been described above, it shouldbe understood that they have been presented by way of example only, andnot limitation. Thus, the breadth and scope of the present disclosureshould not be limited by any of the above-described exemplaryembodiments, but should instead be defined only in accordance with thefollowing claims and their equivalents.

REFERENCES

-   1. Ashburner, J. and K. J. Friston, Voxel-based morphometry—the    methods. Neuroimage, 2000. 11(6 Pt 1): p. 805-21.-   2. Davatzikos, C., Why voxel-based morphometric analysis should be    used with great caution when characterizing group differences.    Neuroimage, 2004. 23(1): p. 17-20.-   3. Aljabar, P., R. A. Heckemann, A. Hammers, J. V. Hajnal, and D.    Rueckert, Multi-atlas based segmentation of brain images: atlas    selection and its effect on accuracy. Neuroimage, 2009. 46(3): p.    726-38.-   4. Heckemann, R. A., J. V. Hajnal, P. Aljabar, D. Rueckert, and A.    Hammers, Automatic anatomical brain MRI segmentation combining label    propagation and decision fusion. Neuroimage, 2006. 33(1): p. 115-26.-   5. Artaechevarria, X., A. Munoz-Barrutia, and C. Ortiz-de-Solorzano,    Combination strategies in multi-atlas image segmentation:    application to brain MR data. IEEE Trans Med Imaging, 2009.    28(8): p. 1266-77.-   6. Rohlfing, T., R. Brandt, R. Menzel, and C. R. Maurer, Jr.,    Evaluation of atlas selection strategies for atlas-based image    segmentation with application to confocal microscopy images of bee    brains. Neuroimage, 2004. 21(4): p. 1428-42.-   7. Langerak, T. R., U. A. van der Heide, A. N. Kotte, M. A.    Viergever, M. van Vulpen, and J. P. Pluim, Label fusion in    atlas-based segmentation using a selective and iterative method for    performance level estimation (SIMPLE). IEEE Trans Med Imaging, 2010.    29(12): p. 2000-8.-   8. Lotjonen, J. M., R. Wolz, J. R. Koikkalainen, L. Thurfi ell, G.    Waldemar, H. Soininen, and D. Rueckert, Fast and robust multi-atlas    segmentation of brain magnetic resonance images. Neuroimage, 2010.    49(3): p. 2352-65.-   9. Warfield, S. K., K. H. Zou, and W. M. Wells, Simultaneous truth    and performance level estimation (STAPLE): an algorithm for the    validation of image segmentation. IEEE Trans Med Imaging, 2004.    23(7): p. 903-21.-   10. Mai, J., G. Paxinos, and T. Voss, Atlas of Human Brain2007, San    Diego: Academic Press.-   11. Puelles, L., M. Harrison, G. Paxinos, and C. Watson, A    developmental ontology for the mammalian brain based on the    prosomeric model. Trends Neurosci, 2013. 36(10): p. 570-8.-   12. Landman, B. A., A. J. Huang, A. Gifford, D. S. Vikram, I. A.    Lim, J. A. Farrell, J. A. Bogovic, J. Hua, M. Chen, S. Jarso, S. A.    Smith, S. Joel, S. Mori, J. J. Pekar, P. B. Barker, J. L. Prince,    and P. C. van Zijl, Multi-parametric neuroimaging reproducibility: a    3-T resource study. Neuroimage, 2011. 54(4): p. 2854-66.-   13. Research consent for cognitively impaired adults:    recommendations for institutional review boards and investigators.    Alzheimer Dis Assoc Disord, 2004. 18(3): p. 171-5.-   14. Mielke, M. M., N. A. Kozauer, K. C. Chan, M. George, J.    Toroney, M. Zerrate, K. Bandeen-Roche, M. C. Wang, P. Vanzijl, J. J.    Pekar, S. Mori, C. G. Lyketsos, and M. Albert, Regionally-specific    diffusion tensor imaging in mild cognitive impairment and    Alzheimer's disease. Neuroimage, 2009. 46(1): p. 47-55.-   15. McKhann, G., D. Drachman, M. Folstein, R. Katzman, D. Price,    and E. M. Stadlan, Clinical diagnosis of Alzheimer's disease: report    of the NINCDS-ADRDA Work Group under the auspices of Department of    Health and Human Services Task Force on Alzheimer's Disease.    Neurology, 1984. 34(7): p. 939-44.-   16. Jack, C. R., Jr., M. A. Bernstein, N. C. Fox, P. Thompson, G.    Alexander, D. Harvey, B. Borowski, P. J. Britson, L. W. J, C.    Ward, A. M. Dale, J. P. Felmlee, J. L. Gunter, D. L. Hill, R.    Killiany, N. Schuff, S. Fox-Bosetti, C. Lin, C. Studholme, C. S.    DeCarli, G. Krueger, H. A. Ward, G. J. Metzger, K. T. Scott, R.    Mallozzi, D. Blezek, J. Levy, J. P. Debbins, A. S. Fleisher, M.    Albert, R. Green, G. Bartzokis, G. Glover, J. Mugler, and M. W.    Weiner, The Alzheimer's Disease Neuroimaging Initiative (ADNI): Mill    methods. J Magn Reson Imaging, 2008. 27(4): p. 685-91.-   17. Faria, A. V., J. Y. Zhang, K. Oishi, X. Li, H. Y. Jiang, K.    Akhter, L. Hermoye, S. K. Lee, A. Hoon, E. Stashinko, M. I.    Miller, P. C. M. van Zijl, and S. Mori, Atlas-based analysis of    neurodevelopment from infancy to adulthood using diffusion tensor    imaging and applications for automated abnormality detection.    Neuroimage, 2010. 52(2): p. 415-428.-   18. Mori, S., K. Oishi, H. Jiang, L. Jiang, X. Li, K. Akhter, K.    Hua, A. V. Faria, A. Mahmood, R. Woods, A. W. Toga, G. B.    Pike, P. R. Neto, A. Evans, J. Zhang, H. Huang, M. I. Miller, P. van    Zijl, and J. Mazziotta, Stereotaxic white matter atlas based on    diffusion tensor imaging in an ICBM template. Neuroimage, 2008.    40(2): p. 570-82.-   19. Oishi, K., A. Faria, H. Jiang, X. Li, K. Akhter, J. Zhang, J. T.    Hsu, M. I. Miller, P. C. van Zijl, M. Albert, C. G. Lyketsos, R.    Woods, A. W. Toga, G. B. Pike, P. Rosa-Neto, A. Evans, J. Mazziotta,    and S. Mori, Atlas-based whole brain white matter analysis using    Large Deformation Diffeomorphic Metric Mapping: Application to    normal elderly and Alzheimer's disease participants. Neuroimage,    2009.-   20. Djamanakova, A., A. V. Faria, J. Hsu, C. Ceritoglu, K.    Oishi, M. I. Miller, A. E. Hillis, and S. Mori, Diffeomorphic brain    mapping based on T1-weighted images: Improvement of registration    accuracy by multichannel mapping. Journal of Magnetic Resonance    Imaging, 2013. 37(1): p. 76-84.-   21. Ceritoglu, C., K. Oishi, X. Li, M. C. Chou, L. Younes, M.    Albert, C. Lyketsos, P. C. van Zijl, M. I. Miller, and S. Mori,    Multi-contrast large deformation diffeomorphic metric mapping for    diffusion tensor imaging. Neuroimage, 2009.-   22. Christensen, G. E., S. C. Joshi, and M. I. Miller, Volumetric    transformation of brain anatomy. IEEE Trans Med Imaging, 1997.    16(6): p. 864-77.-   23. Tang, X., K. Oishi, A. V. Faria, A. E. Hillis, M. Albert, S.    Mori, and M. I. Miller, Bayesian parameter estimation and    segmentation in the multi-atlas random orbit model. PLOS ONE, 2013.    in press.-   24. Tang, X., K. Oishi, A. V. Faria, A. E. Hillis, M. S. Albert, S.    Mori, and M. I. Miller, Bayesian Parameter Estimation and    Segmentation in the Multi-Atlas Random Orbit Model. PloS One, 2013.    8(6): p. e65591.-   25. Djamanakova, A., Tang, X., Li, X., Faria, A. V., Ceritoglu, C.,    Oishi, K., et al. (2014). Tools for multiple granularity analysis of    brain MRI data for individualized image analysis. Neuroimage 101,    168-176. doi: 10.1016/j.neuroimage.2014.06.046.-   26. Joshi, S., and Miller, M. I. (2000). Landmark Matching via Large    Deformation Diffeomorphisms. IEEE Trans Image Processing 9(8),    1357-1370.-   27. Tang, X., Crocetti, D., Kutten, K., Ceritoglu, C., Albert, M.    S., Mori, S., et al. (2015). Segmentation of brain magnetic    resonance images based on multi-atlas likelihood fusion: testing    using data with a broad range of anatomical and photometric    profiles. Front Neurosci 9, 61. doi: 10.3389/fnins.2015.00061.-   28. Wu, D., Ma, T., Ceritoglu, C., Li, Y., Chotiyanonta, J., Hou,    Z., et al. (2015). Resource atlases for multi-atlas brain    segmentations with multiple ontology levels based on T1-weighted MRL    Neuroimage 125, 120-130. doi: 10.1016/j.neuroimage.2015.10.042.

What is claimed is:
 1. A computer-implemented method of constructinghuman-readable sentences from imaging data of a subject, comprising:receiving imaging data comprising a plurality of image elements of atleast one region of interest of the subject; segmenting, using at leastone data processor, the imaging data of said region of interest into aplurality of sub-regions, each sub-region comprising a portion of saidplurality of image elements; calculating an abnormality factor for eachof the sub-regions by quantitatively analyzing segmented imageinformation of said imaging data of said sub-regions using data from anormal database; comparing each abnormality factor to a threshold value;constructing a human-understandable sentence for the subject when acorresponding abnormality factor exceeds the threshold, eachhuman-understandable sentence referencing a physical structure thresholdassociated with the calculation for the region or sub-region; andoutputting the human-understandable sentences for the at least oneregion of the subject.
 2. The method of claim 1, further comprisinggenerating additional structures by analyzing multiple levels ofgranularity for the segmented sub-regions, wherein calculating theabnormality factors includes calculating an abnormality factor for eachof the additional structures.
 3. The method of claim 2, furthercomprising determining clinically relevant structures using a clinicalknowledge database from the segmented and generated structures, whereinconstructing said human-understandable sentence includes incorporatingthe clinically relevant structures.
 4. The method of claim 3, whereinconstructing said human-understandable sentence takes into accountrelationships among clinically relevant structures.
 5. The method ofclaim 4, wherein the relationships are nested.
 6. The method of claim 1,further comprising analyzing said abnormality factor that exceeds saidthreshold of said subject according to one of a plurality of predefinedseverity thresholds, wherein each of the human-understandable sentencereferences a physical structure and a severity threshold associated withthe calculation for the region or sub-region.
 7. The method of claim 1,wherein said calculating said abnormality factor includes calculating anabnormality factor for each sub-region that exceeds said threshold. 8.The method of claim 1, wherein said human-understandable sentence isconstructed using a set of predetermined rules based on a relationshipbetween a size of a structure and a size of a corresponding at least onesub-region having an abnormality factor.
 9. The method of claim 1,wherein the calculated abnormality factor is based on calculatingstatistical significance from averages and standard deviations ofage-matched control subject data.
 10. The method of claim 1, wherein theimage information is at least one of size or intensity of the imagingdata.
 11. The method of claim 10, wherein the quantitatively analyzingsaid image information includes measuring differences between said sizeor intensity of said imaging data of said at least one region withreference imaging data.
 12. The method of claim 10, wherein thecomparing takes place based on a predetermined relationship between asize or intensity of the imaging data and a clinical diagnosis fallingwithin a statistically significant range.
 13. The method of claim 12,wherein the comparing takes place based on non-image clinicalinformation.
 14. The method of claim 1, wherein the imaging data isgenerated from at least one of magnetic resonance imaging (MRI),computed tomography (CT), positron emission tomography (PET),ultrasound, or nuclear tracer three-dimensional imaging.
 15. The methodof claim 1, wherein said segmenting includes segmenting the imaging dataof said region of interest into a plurality of sub-regions at aplurality of levels of granularity, the plurality of levels ofgranularity having a relationship such that a level of granularity hasfewer structures at a lower level of granularity, and wherein saidcalculating includes calculating an abnormality factor at each of theplurality of levels of granularity.
 16. The method of claim 15, whereinsaid relationship is based on the sizes and/or intensities of a singlestructure or combinations of multiple segmented structures.
 17. Themethod of claim 16, wherein the sizes and/or intensities of multiplestructures is combined by Boolean and/or arithmetic operators toconstruct an elaborated relationship between the outputted sentences andanatomical features.
 18. The method of claim 17, wherein therelationship between the anatomical features and the outputted sentencesis further elaborated by segmenting the imaging data at a plurality oflevels of granularity, the plurality of levels of granularity having arelationship such that a level of granularity has fewer structures at alower level of granularity, and wherein said calculating includescalculating an abnormality factor at each of the plurality of levels ofgranularity.
 19. The method of claim 1, further comprising: mapping aplurality of abnormality factors to a plurality of predeterminedclinical diagnoses in a database on a data storage device; and providinga clinical diagnosis of the subject based on a correlation between thestored clinical diagnoses and the outputted sentences of the subject.20. The method of claim 1, further comprising calculating a globalabnormality factor for the imaging data of the at least one region ofinterest by quantitatively analyzing global image information of saidimaging data of said at least one region of interest, wherein comparingsaid abnormality factor includes cataloguing said compared abnormalityfactor and said global abnormality factor of said subject based on saidcalculating steps according to one of a plurality of predefined severitythresholds.
 21. The method of claim 20, wherein said calculating saidglobal abnormality factor comprises warping said imaging data toreference imaging data and calculating a difference.
 22. The method ofclaim 20, further comprising for the outputted sentences that have aclinically meaningful significance, reconstructing a relationshipbetween the clinically meaningful sentences and the global and segmentedimage information.
 23. A computer system for constructing human-readablesentences from imaging data of a subject, comprising: a memorycomprising computer-executable instructions; and a data processor thatis coupled to the memory, said data processor being configured toexecute the computer-executable instructions to: receive imaging datacomprising a plurality of image elements of at least one region ofinterest of the subject; segment the imaging data of said region ofinterest into a plurality of sub-regions, each sub-region comprising aportion of said plurality of image elements; calculate an abnormalityfactor for each of the sub-regions by quantitatively analyzing segmentedimage information of said imaging data of said sub-regions using datafrom a normal database; compare each abnormality factor to a thresholdvalue; construct a human-understandable sentence for the subject when acorresponding abnormality factor exceeds the threshold, eachhuman-understandable sentence referencing a physical structureassociated with the calculation for the region or sub-region; and outputthe human-understandable sentences for the at least one region of thesubject.
 24. A non-transitory computer-readable medium for constructinghuman-readable sentences from imaging data of a subject, thecomputer-readable medium having instructions that, when executed by atleast one data processor, cause a computing system to: receive imagingdata comprising a plurality of image elements of at least one region ofinterest of the subject; segment the imaging data of said region ofinterest into a plurality of sub-regions, each sub-region comprising aportion of said plurality of image elements; calculate an abnormalityfactor for each of the sub-regions by quantitatively analyzing segmentedimage information of said imaging data of said sub-regions using datafrom a normal database; compare each abnormality factor to a thresholdvalue; construct a human-understandable sentence for the subject when acorresponding abnormality factor exceeds the threshold, eachhuman-understandable sentence referencing a physical structureassociated with the calculation for the region or sub-region; and outputthe human-understandable sentences for the at least one region of thesubject.